check if llama.cpp is feasible to be used directly instead of llama-cpp-python

[kyteinsky]

for the docker builds, there are some limitations added by llama-cpp-python which prevent us from building llama.cpp with flexible support for SSE, AVX(2) and AVX512. They are not loaded on demand but hardcoded to support AVX2 or fail. If AVX512 were to be compiled (GGML_AVX512=ON), that build would fail on all systems that don't support it even if the system support AVX2 and the build has that compiled in.
the llama.cpp flags for flexible/dynamic build: GGML_BACKEND_DL=ON GGML_CPU_ALL_VARIANTS=ON
^ this may be addressed later on but not a priority.

for now the previous requirement of AVX2 has been kept intact.

Originally posted by @kyteinsky in #295 (comment)

[marcelklehr]

1. Are we still using llama-cpp-python at all in here? I thought we're using llama.cpp server?
2. Is that something that is on the roadmap for llama-cpp-python, perhaps?

[kyteinsky]

1. yep, llama-cpp-python server :)
   

   context_chat_backend/main_em.py

   Lines 154 to 181 in fe0c3a2

   	from llama_cpp.server.app import create_app
   	from llama_cpp.server.settings import ModelSettings, ServerSettings
   	base_url = urlparse(em_conf.base_url)
   	host = base_url.hostname or '127.0.0.1'
   	port = base_url.port or 5000
   	server_settings = ServerSettings(host=host, port=port)
   	model_settings = [ModelSettings(model_alias=DEFAULT_EM_MODEL_ALIAS, embedding=True, **em_conf.llama)]
   	app = create_app(
   	server_settings=server_settings,
   	model_settings=model_settings,
   	)
   	uv_log_config = uvicorn.config.LOGGING_CONFIG # pyright: ignore[reportAttributeAccessIssue]
   	use_colors = False if k8s_env else (app_config.use_colors and os.getenv('CI', 'false') == 'false')
   	if k8s_env:
   	uv_log_config['formatters']['default'] = logging_config['formatters']['json']
   	uv_log_config['formatters']['access'] = logging_config['formatters']['json']
   	else:
   	uv_log_config['formatters']['json'] = logging_config['formatters']['json']
   	uv_log_config['handlers']['file_json'] = logging_config['handlers']['file_json']
   	uv_log_config['loggers']['uvicorn']['handlers'].append('file_json')
   	uv_log_config['loggers']['uvicorn.access']['handlers'].append('file_json')
   	uvicorn.run(
   	# todo: use string import of the app
   	app=app,

2. couldn't find anything like that, not in the issues either as far as I looked.

Also, it seems features of llama.cpp need to be integrated/connected in the python wrapper to make use of them, it doesn't come free so they are delayed/mismatched in the python wrapper.
things like: #218 (comment)