Analysing FM20 with Python

[CowShedBarmyArmy]

Analysing FM with Python

I know MS Excel has become very popular on here for analysing squads etc but does anyone use Python? Its very powerful and once you have the code setup you can create charts and tables directly from FM20 without any manual entry required.

I've been learning it while working from home during the pandemic and find it easier to use than excel for squad analysis and analysing scouting targets.

 

What it does:

I've been using it to rate my squad by DNA score and position.

For each position in my current tactic I have a filtered table and scatter plot. I use these to assess my current squad and players found in the player search to see who would fit in to my style.

I also analyse

Club DNA

Here is an example of the code to show how easy it is to edit.

How to use it:

I created two squad views:

       One with all attributes                                                                                                            One with all stats.

    

 

I also create the same view in the scouting module so I can assess players from my searches. I still need to create my stats one.

I export these (ctrl-p  ---  print to webpage) to a nominated folder on my HDD with a fixed file name.

(1) Install anaconda which is open source and free for single home users:

https://www.anaconda.com/products/individual

This contains Jupyter Notebooks which I used to enter my code.

(2) You will need to install the following packages

numpy, pandas, plotly, cufflinks

Here is how to do that:

https://docs.anaconda.com/anaconda/user-guide/tasks/install-packages/

Edit the paths to the html files from FM20

Once I run my file in Jupyter, It imports all the data from the .html files and populates my tables and graphs automatically. It takes a few seconds and I have lots of data to play with.

I'm still pretty basic at the moment but happy to share code etc with anyone or take advice from anyone who knows what they are doing.

 

 

 

 

 

 

Edited May 10, 2020 by CowShedBarmyArmy
Adding info

[Guest]

Nice staff here!

[Guest]

@Giggiux interested to see that @CowShedBarmyArmy

[gam945]

Always wanted to do something similar and doing the "analysis" job of the manager IRL, for example monitoring the squad during the season with stats and attributes...but laziness always takes over

Planning of beginning a new long term save soon, I'll definitively try to be serious about it lol

Also this could be a very useful tool for scouting hundreds of players stats

Will share here if I get something done

[CowShedBarmyArmy]

I'm happy to share my code and view etc to get you started if you want. This is just for fun for me, not looking to profit.

[22764636]

I have compiled a python script that tells me what's the best role-duty for a player given his attributes

[CowShedBarmyArmy]

12 minutes ago, 22764636 said:

I have compiled a python script that tells me what's the best role-duty for a player given his attributes

Nice, one. Good to know there's another nerd out there!

Feel free to share your stuff on here.

[PenguinKenny]

2 hours ago, 22764636 said:

I have compiled a python script that tells me what's the best role-duty for a player given his attributes

I'd be interested in seeing this.

[CowShedBarmyArmy]

Here is  a link to the code if anyone is interested.

https://github.com/winston1984-sys/FM20-Analysis

[Egecann]

I’m gonads definitely learn python during quarantine. Thanks for the code btw

[CowShedBarmyArmy]

6 hours ago, Egecann said:

I’m gonads definitely learn python during quarantine. Thanks for the code btw

No worries, give me a shout if you need any help.

[gam945]

@CowShedBarmyArmy wondering how do you visualize your plots?

EDIT: Will just go with Jupyter Lab/Notebook, less of a hassle and it would be sort of a report

I have defined all the plots and data I wanna see using my excel tool (which I can share):
It contains a dictionnary for the attributes (long name, short name), the stats (long & short name, categorization also) a sheet to define my tactic and a sheet with a macro that converts multiple cells into a string containing their values into a python list format, for example ['aaa','bbb','ccc']

I've already wrote a code to import data and clean it, for example converting appropriate stats to integer of float type, or converting appearances (x) y into x+y.

An example of simple plot with matplotlib:

 

In the long term, I also project to analyse how attributes influence my KPIs, so projecting to do some regression analysis after setting up my dashboard, would be a cool project

Edited May 10, 2020 by gam945

[CowShedBarmyArmy]

 

19 hours ago, gam945 said:

@CowShedBarmyArmy wondering how do you visualize your plots?

EDIT: Will just go with Jupyter Lab/Notebook, less of a hassle and it would be sort of a report

I have defined all the plots and data I wanna see using my excel tool (which I can share):
It contains a dictionnary for the attributes (long name, short name), the stats (long & short name, categorization also) a sheet to define my tactic and a sheet with a macro that converts multiple cells into a string containing their values into a python list format, for example ['aaa','bbb','ccc']

I've already wrote a code to import data and clean it, for example converting appropriate stats to integer of float type, or converting appearances (x) y into x+y.

An example of simple plot with matplotlib:

 

In the long term, I also project to analyse how attributes influence my KPIs, so projecting to do some regression analysis after setting up my dashboard, would be a cool project

Hi gam945,

I use Plotly at the moment, couldnt quite work out how to add the player name to a scatter plot on matplotlib.

https://plotly.com/python/plotly-express/

Jupyter is very easy to use so I would definately recommend it. It allows you to test small chunks of code at a time.

I'm impressed with the clarity of your plan. I planned to create this kind of detailed plan up front but I just ending up diving straight in and I've been winging it ever since.

I cant wait to see your progress.

 

[WhoAreYa?]

This is fantastic! The Python aspect is perhaps beyond me but your statistical analysis is very much up my street. I've just started a save with Brentford FC who are massive on stats in real life and it is definitely an area that I'm going to be placing a lot of emphasis on, in the transfer market especially. Football seems to be becoming more analytical by the day and, honestly, it surprises me that more people don't look at the in-game stats which are actually quite numerous now - and will only become more so. There are some that I find very important such as save % for keepers, shots on target for strikers and key passes.

 

What are your favourite stats and how do these change by position or by role? For example what stats do you place most emphasis on for say central defenders? 

 

Edited May 12, 2020 by WhoAreYa?

[CowShedBarmyArmy]

Thanks for the nice feedback. Maybe one day I will be competent enough to make the program really easy to use!

I haven't finalised my stats analysis yet as I have been developing the python code but for central defenders I am currently using : Interceptions/90 vs Headers won/90

I plan to use the stats in here https://totalfootballanalysis.com/category/data-analysis which will mean I have to create some custom ones.

Work has been busy today so havent done much last two days but I will soon.

 

Edited May 12, 2020 by CowShedBarmyArmy

[WhoAreYa?]

This may be of interest to you, @CowShedBarmyArmy?

https://theresonlyoneball.com/2019/12/31/0104-scouting-and-transfers/ from  @fmFutbolManager

As I mentioned with my Brentford save, this is the kind of thing that I want to be setting up. Fascinating stuff! 

[FMFutbol]

Will try this later!

[CowShedBarmyArmy]

21 hours ago, WhoAreYa? said:

This may be of interest to you, @CowShedBarmyArmy?

https://theresonlyoneball.com/2019/12/31/0104-scouting-and-transfers/ from  @fmFutbolManager

As I mentioned with my Brentford save, this is the kind of thing that I want to be setting up. Fascinating stuff! 

Thanks for the link, I've read his posts before and they are excellent. I will try and integrate some ideas into my stuff.

I've made some progress today with converting data etc so it should be full steam ahead now.

Think I have a decent way to assess goalkeepers:

Saves/90 vs Goals Conceded/90

Theory being a keeper making lots of saves in a team that doesnt concede much must be doing ok.

I'd love total shots faced but I can't find that in FM20. I had to calculate Saves/90 myself as it doesn't exist.

If you have specific metrics you'd like then let me know and i'll create them and upload the file.

Edited May 14, 2020 by CowShedBarmyArmy

[CowShedBarmyArmy]

17 hours ago, FMFutbol said:

Will try this later!

If you need help give me a shout. I'm slowly going mad creating this stuff but I'm 6 games from an unbeaten season with 1860 Munich so it must be working

[CowShedBarmyArmy]

On 10/05/2020 at 01:15, Egecann said:

I’m gonads definitely learn python during quarantine. Thanks for the code btw

I'm uploading new versions all the time so keep checking!

[WhoAreYa?]

1 hour ago, CowShedBarmyArmy said:

Thanks for the link, I've read his posts before and they are excellent. I will try and integrate some ideas into my stuff.

I've made some progress today with converting data etc so it should be full steam ahead now.

Think I have a decent way to assess goalkeepers:

Saves/90 vs Goals Conceded/90

Theory being a keeper making lots of saves in a team that doesnt concede much must be doing ok.

I'd love total shots faced but I can't find that in FM20. I had to calculate Saves/90 myself as it doesn't exist.

If you have specific metrics you'd like then let me know and i'll create them and upload the file.

I seem to remember that there's a stat shown on a player's data profile that isn't shown on the search filter. Their personal data report card near where you look at their coach report. Sure it's something like shots faced or saves made. I can't check right now as I'm waiting for a new computer and can't quite remember it! 

[CowShedBarmyArmy]

Two new files uploaded to GitHub.

Added message box to choose between Current Squad data or Potential Signings. This saves having too much data on screen at one time.

Now the coding is in reasonable shape (for a rookie), I'll focus on creating new player profile scatter plots. At the moment I only have:

Goalkeepers - Saves/90 v Goals Conceded/90

Best Tacklers - Tackles/90 v Tackle Won %

Best Headers - Headers Won/90 v Header Win %

Ball Winners - Tackles/90 v Interceptions/90

Wing Play - Dribbles Per Game v Assists/90

Playmakers - Key Passes/90 v Chances Created/90

Goalscorers - Goals/90 v Shot On Target %

[CowShedBarmyArmy]

On 14/05/2020 at 20:50, WhoAreYa? said:

I seem to remember that there's a stat shown on a player's data profile that isn't shown on the search filter. Their personal data report card near where you look at their coach report. Sure it's something like shots faced or saves made. I can't check right now as I'm waiting for a new computer and can't quite remember it! 

Think i've sussed it out.

shots saved + goals conceded = shots faced.

Cheers for the heads up

 

Edited May 19, 2020 by CowShedBarmyArmy

[nick1408]

Any chance you could work out xG with this? I'm quite interested to try it out

 

https://dictatethegame.com/2019/09/27/getting-expected-goals-or-xg-into-fm19/

[CowShedBarmyArmy]

Apart from method one they require manual calculations so I don't think I can achieve this in python.

I can try the quick and dirty one though.

[Guest]

9 hours ago, nick1408 said:

Any chance you could work out xG with this? I'm quite interested to try it out

 

https://dictatethegame.com/2019/09/27/getting-expected-goals-or-xg-into-fm19/

This must come with the game IMHO.

[FMFutbol]

On 15/05/2020 at 17:47, CowShedBarmyArmy said:

Two new files uploaded to GitHub.

Added message box to choose between Current Squad data or Potential Signings. This saves having too much data on screen at one time.

Now the coding is in reasonable shape (for a rookie), I'll focus on creating new player profile scatter plots. At the moment I only have:

Goalkeepers - Saves/90 v Goals Conceded/90

Best Tacklers - Tackles/90 v Tackle Won %

Best Headers - Headers Won/90 v Header Win %

Ball Winners - Tackles/90 v Interceptions/90

Wing Play - Dribbles Per Game v Assists/90

Playmakers - Key Passes/90 v Chances Created/90

Goalscorers - Goals/90 v Shot On Target %

I like the 3d plotly express charts the best as you can include more information. 
Like for wing play, dribbles per game, cross completion, pass percentage.

Feels like I get better information with 3d than just 2d. 

Looked at trying a 3d surface plot but not sure how to fit it to the data

[CowShedBarmyArmy]

On 27/05/2020 at 23:52, FMFutbol said:

I like the 3d plotly express charts the best as you can include more information. 
Like for wing play, dribbles per game, cross completion, pass percentage.

Feels like I get better information with 3d than just 2d. 

Looked at trying a 3d surface plot but not sure how to fit it to the data

I haven't tried these yet, great idea thanks.

[calv84]

Hi guys

 

I've been trying to teach myself Python over the last few weeks and fm seemed to be a good place to start, and this thread has been a big inspiration for me and I refer back to it every couple days.

 

My goal is to ultimately unearth good value players from scouting only in game statistics not attributes. 

 

An issue that i have run into is converting the Value string into a usable float.  The way the value data is extracted from the game makes it incredibly difficult (for my simple mind) to apply a set of rules to convert it into usable data.

 

Hopefully below will highlight the issue i'm running into.

 

The way I had been looking to take the issue is to replace K or M with a set number of 0's and remove the non-numeric characters. This is obviously isn't going to work due to the structure of the exported data. For example if i replaced M with 000000 i would receive: 

• 2m = 2000000
• 7.5m = 75000000
• 9.75m = 975000000

 

Can anyone tell me how they have solved this? Or if there is a different field available? I'm really not keen on this being a manual process. 

 

 

[andershs]

1 hour ago, calv84 said:

Hi guys

 

I've been trying to teach myself Python over the last few weeks and fm seemed to be a good place to start, and this thread has been a big inspiration for me and I refer back to it every couple days.

 

My goal is to ultimately unearth good value players from scouting only in game statistics not attributes. 

 

An issue that i have run into is converting the Value string into a usable float.  The way the value data is extracted from the game makes it incredibly difficult (for my simple mind) to apply a set of rules to convert it into usable data.

 

Hopefully below will highlight the issue i'm running into.

 

The way I had been looking to take the issue is to replace K or M with a set number of 0's and remove the non-numeric characters. This is obviously isn't going to work due to the structure of the exported data. For example if i replaced M with 000000 i would receive: 

• 2m = 2000000
• 7.5m = 75000000
• 9.75m = 975000000

 

Can anyone tell me how they have solved this? Or if there is a different field available? I'm really not keen on this being a manual process. 

 

 

 

This is just from the top of my head.

But couldn't you do something like this:

If you encounter 'm' and you know you are dealing with monetary values:

1. Remove 'm' from the string or trim the string and set the string length to n-1
2. Try to cast the string to float
3. Multiply the float by 1000000

This can similarly also be applied to 'k' monetary values (multiplying instead with 100000).

I know there are some steps and each step could present problems, but it is worth a try.

Edited October 11, 2020 by andershs

[calv84]

1 hour ago, andershs said:

 

This is just from the top of my head.

But couldn't you do something like this:

If you encounter 'm' and you know you are dealing with monetary values:

1. Remove 'm' from the string or trim the string and set the string length to n-1
2. Try to cast the string to float
3. Multiply the float by 1000000

This can similarly also be applied to 'k' monetary values (multiplying instead with 100000).

I know there are some steps and each step could present problems, but it is worth a try.

Of course...

 

I'll give that a go. Thank you. 

[EnigMattic1]

Damn. I really wish I found this thread months ago I actually have the stuff on my laptop to install Python but never got round to it. I kinda wish I had now. Instead, I manually did all the formulas in excel for my club dna AND every position available. 🙄

I always got too confused when it came to some of the stuff like tackles per 90 and tackle win %, so I just didn't bother with that, despite wanting to. 

Edited October 11, 2020 by EnigMattic1

[CowShedBarmyArmy]

On 11/10/2020 at 08:42, calv84 said:

Hi guys

 

I've been trying to teach myself Python over the last few weeks and fm seemed to be a good place to start, and this thread has been a big inspiration for me and I refer back to it every couple days.

 

My goal is to ultimately unearth good value players from scouting only in game statistics not attributes. 

 

An issue that i have run into is converting the Value string into a usable float.  The way the value data is extracted from the game makes it incredibly difficult (for my simple mind) to apply a set of rules to convert it into usable data.

 

Hopefully below will highlight the issue i'm running into.

 

The way I had been looking to take the issue is to replace K or M with a set number of 0's and remove the non-numeric characters. This is obviously isn't going to work due to the structure of the exported data. For example if i replaced M with 000000 i would receive: 

• 2m = 2000000
• 7.5m = 75000000
• 9.75m = 975000000

 

Can anyone tell me how they have solved this? Or if there is a different field available? I'm really not keen on this being a manual process. 

 

 

I can send you my code to show you how i solved this.

financial = squad[['Name', 'Age', 'Wage', 'Value', 'Expires', 'Mins']].copy()
for column in ['Value', 'Wage']:
    financial[column] = financial[column].str.replace('Â', '')
    financial[column] = financial[column].str.replace('p/w', '')
    financial[column] = financial[column].str.replace('£', '').copy()

Kind of weird out of context, but hopefully makes sense.

[CowShedBarmyArmy]

On 12/10/2020 at 00:15, EnigMattic1 said:

Damn. I really wish I found this thread months ago I actually have the stuff on my laptop to install Python but never got round to it. I kinda wish I had now. Instead, I manually did all the formulas in excel for my club dna AND every position available. 🙄

I always got too confused when it came to some of the stuff like tackles per 90 and tackle win %, so I just didn't bother with that, despite wanting to. 

If I can help at all give me a shout. I was a complete novice when I started. I still don't fully understand all of what I'm doing but you make big gains by diving in and having a go.

[CowShedBarmyArmy]

My code is an absolute shambles at the moment as I have been trying all sorts of things and been busy in work but feel free to poke around and have a look.

I've uploaded a squad view and scouting view for exporting the data. Also a css file to make the html output look prettier.

All you need to do is set the correct path so the program can find your exported fm data (change bold text to your location)

 

squad = pd.read_html(
    r"C:\Users\stuar\Documents\Sports Interactive\Football Manager 2020\html\fmi_4_sq.html"
 )[0]
scouting = pd.read_html(
    r"C:\Users\stuar\Documents\Sports Interactive\Football Manager 2020\html\fmi_4_sc.html"
 )[0]

 

 

fmi_4.py df_style.css scout_analysis_fmi4.fmf squad_analysis_fmi_4.fmf

Edited November 8, 2020 by CowShedBarmyArmy