Mistral, best open-source AI so far, and it's from Europe, 7B model tested, full-transcript.

Mistral, ein kleines Startup aus Frankreich, das OpenAI Konkurrenz macht, obwohl klein nicht mehr ganz zutreffend ist. Mistral hat sich im Moment zum führenden Open-Source-Anbieter von Sprachmodellen entwickelt. Und das aus Europa.

Und auch die Bewertung des Startups ist beachtlich und hat bereits die Milliarden-Dollar Grenze überschritten und auch Microsoft ist zunehmend an Mistral interessiert und investiert bereits in das Unternehmen, würde es wahrscheinlich am liebsten aufkaufen.

Heute testen wir das kleinere der beiden Open-Source-Sprachmodelle, die Mistral veröffentlicht hat.

Mistral7B hat die KI-Community mit seiner Größe und Performance überrascht.

Das Modell ist ein eigenständig trainiertes LLM, das nicht auf Meta's Llama basiert und schlägt in der 7B Variante das größere Llama2 13B Modell und kommt gefühlt sogar an das Niveau von ChatGPT (GPT-3.5) heran, was unglaublich für so ein kleines Modell ist.

Der MMLU-Score von Mistral7B erreicht 60.1 (GPT-3.5 70, GPT-4 86.5).

Die 4-Bit Quantisierung benötigt für die Interference in etwa 10 GB VRAM und vorausgesetzt man hat genug VRAM, wird eine Antwort in nur 2-3 Sekunden generiert. Alles lokal ohne Zensur.

Das ist der Vorteil von kleinen Open-Source-Modellen.

Was sagt ihr dazu? Wie findet ihr die Antworten von Mistral7B?

Open-Source LLMs. AI-generated illustration (Bing)

English

Mistral, a small startup from Paris, France that is competing with OpenAI, although small is no longer quite accurate. Mistral is currently the leading open source provider of language models. And that from Europe.

The startup's valuation is also impressive and has already exceeded the billion dollar mark. Microsoft is also increasingly interested in Mistral and is already investing in the company, and would probably prefer to buy it up.

Today we are testing the smaller of the two open-source language models that Mistral has released.

Mistral7B has surprised the AI community with its small size and performance.

The model is a standalone trained LLM that is not based on Meta's Llama and the 7B variant beats the larger Llama2 13B model and feels like it even comes close to the level of ChatGPT (GPT-3.5), which is incredible for such a small model.

The MMLU score of Mistral7B reaches 60.1 (GPT-3.5 70, GPT-4 86.5).

The 4-bit quantization requires about 10 GB of VRAM for the interference and provided you have enough VRAM, a response is generated in just 2-3 seconds. All locally without censorship.

This is the advantage of small open-source models.

What do you think? How do you like Mistral7B's answers?

Mistral 7B Instruct v0.2

https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GPTQ

https://mistral.ai/technology/#models

https://mistral.ai/news/announcing-mistral-7b/

Quantization: Mistral-7B-Instruct-v0.2-GPTQ (main)
Parameters: simple-1
Zero Shot

Interface (oobabooga/text-generation-webui)

https://github.com/oobabooga/text-generation-webui

How good is Mistral7B? Full-Transcript

Q

What is the capital city of Austria? Answer only the name of the city.

A (Mistral 7B)

The capital city of Austria is Vienna.

Result

Correct city, but too long answer.

Q

Is Bitcoin a security or commodity?

A (Mistral 7B)

Based on regulatory definitions and legal rulings in various jurisdictions, Bitcoin and other cryptocurrencies can be considered both a currency (commodity) and a security depending on the specific context and use case. In some instances, like when it's used as a medium of exchange or stored value, it may lean more towards being a commodity. However, when investors expect profits based on others' actions, such as buying Bitcoin as an investment expectation that others will pay more for it in the future, then it could be viewed as a security. It's essential to consult with financial professionals and legal experts for advice tailored to your situation.

Result

Answer a little bit too vague, does not mention SEC not giving clear answer.

Q

What is the hive blockchain?

A (Mistral 7B)

The Hive Blockchain is a decentralized platform that focuses on creating and managing applications using the Hive blockchain technology. It's designed to be more user-friendly compared to other blockchains like Ethereum or EOS. Applications built on Hive are called 'Hives' and they cover various use cases such as social media, blogging, gaming, and more. Each Hive application allows users to earn cryptocurrency rewards by engaging with the community and contributing value through content creation or curation.

Result

Not bad, except that DApps are called "Hives", lol.

Q

What is 2^8?

A (Mistral 7B)

The answer to that mathematical expression is 256.

Result

Correct.

Q

Can you compute the 8th root of 2^8?

A (Mistral 7B)

The 8th root of 2 raised to the power of 8 is a complex number. Its real part is approximately 1.7356, and its imaginary part is approximately 0.9145. This result comes from the mathematical property that the exponentiation of a base with an even exponent greater than 2 results in a complex number when the base is not a real number or not a perfect power.

Result

Incorrect.

Q

Solve the puzzle: When I was 6 my sister was half my age, now i'm 70. How old's my sister? Let's think step by step.

A (Mistral 7B)

To find out your sister's current age based on the information provided, we need to determine how many years she was younger when you were 6, and then add that number to your current age. So, let me calculate that for you...

When you were 6, your sister was half your age, which means she was 3 years old at that time (half of 6). Therefore, her current age would be 3 years less than your current age of 70. So, your sister would currently be around 67 years old.

Result

Correct, but does not found hidden subtlety (2-4 year difference depending on when her sister has her bday and the reference dates).

Q (Bar test)

Answer the following exam question based on US law:

A father lived with his son, who was an alcoholic. When drunk, the son often became violent and physically abused his father. As a result, the father always lived in fear. One night, the father heard his son on the front stoop making loud obscene remarks. The father was certain that his son was drunk and was terrified that he would be physically beaten again. In his fear, he bolted the front door and took out a revolver. When the son discovered that the door was bolted, he kicked it down. As the son burst through the front door, his father shot him four times in the chest, killing him. In fact, the son was not under the influence of alcohol or any drug and did not intend to harm his father. At trial, the father presented the above facts and asked the judge to instruct the jury on self-defense. How should the judge instruct the jury with respect to self-defense?

(A) Give the self-defense instruction, because it expresses the defense’s theory of the case.
(B) Give the self-defense instruction, because the evidence is sufficient to raise the defense.
(C) Deny the self-defense instruction, because the father was not in imminent danger from his son.
(D) Deny the self-defense instruction, because the father used excessive force

What is the correct answer?

A (Mistral 7B)

(B) Give the self-defense instruction, because the evidence is sufficient to raise the defense. However, it's important to note that this is just a legal hypothetical scenario, and actual legal outcomes depend on various factors beyond what's provided here. Additionally, laws may vary by jurisdiction, so it's crucial to consult with a legal professional for specific advice.

Result

Correct, correct answer would be B. I am not a lawyer but assume it could be justified in the US by the Stand-your-ground law.

Q (GMAT test)

Select the best of the answer choices given.

The interview is an essential part of a successful hiring program because, with it job applicants who have personalities that are unsuited to the requirements of the job will be eliminated from consideration.

This argument logically depends on which of the following assumptions?

A: A Hiring program will be successful if it includes interviews.

B: The interview is a more important part of a successful hiring program than is the development of a job description.

C: Interviewers can accurately identify applicants whose personalities are unsuited to the requirements of the job.

D: The only purpose of an interview is to evaluate whether job applicants’ personalities are suited to the requirements of the job.

E: The fit of job applicants’ personalities to the requirements of the job was once the most important factor in making hiring decisions.

A (Mistral 7B)

C. Interviewers can accurately identify applicants whose personalities are unsuited to the requirements of the job. However, it's important to note that the accuracy of such identification can depend on various factors including the effectiveness of the interview process and the skills of the interviewer.

Result

Correct.

Q

Write a hello world program in Commodore 64 Basic that flickers the screen.

A (Mistral 7B)

I'm glad you asked about programming! However, writing a "Hello World" program that flickers the screen on a Commodore 64 might not be straightforward due to the limitations of its BASIC interpreter. Instead, let me suggest a simple "Hello World" program first:

10 PRINT "Hello World"

RUN

As for flickering the screen, it usually involves manipulating the color palette or video memory directly, which goes beyond the capabilities of standard BASIC commands. You may need to use machine language instructions or specialized hardware like the POKEY chip. If you have specific requirements or are interested in learning more about low-level programming on the Commodore 64, feel free to ask

Result

Was not able to produce the screen flickering, although it's possible in Basic, but print statemant was correct.

Conclusion

Super impressive for such a small model with only 7B parameters, super fast and mostly accurate answers.

Live your Secrets and Hive Prosper 🍯

xx Viki @vikisecrets

Posted using STEMGeeks