Compare commits

...

18 Commits

Author SHA1 Message Date
cf21d5cd74 Fixed donwloader to work with direct cookie. Cleaned some files. 2024-09-10 14:50:21 +03:00
7fd7492dc2 Changed file naming 2024-07-09 16:42:14 +03:00
49eb64f589 Addedd Prerequisites to README 2024-07-01 16:34:58 +03:00
cb8939a3e3 Added FLAC Tagging 2024-07-01 16:04:53 +03:00
7c213fe037 Added MP3 tagging 2024-07-01 15:39:42 +03:00
42ab8dbcb1 Added cover resizing 2024-07-01 15:17:40 +03:00
21058e5ac4 Prepare for tagging 2024-07-01 15:10:04 +03:00
01fc3a0551 Added missing cleans 2024-07-01 14:47:21 +03:00
02b282b80f Names are now cleaned for illegal symbols in filenames 2024-07-01 14:46:18 +03:00
f68041aa44 Changed TODO format 2024-07-01 14:25:19 +03:00
0260ec85b6 Updated README 2024-07-01 13:00:06 +03:00
a0cf3766bf Updated README and TODO 2024-07-01 12:58:21 +03:00
911cb8fdf3 Changed README format 2024-07-01 12:54:24 +03:00
438ccdc9d7 Reworked downloaded now available. Tagging not working 2024-07-01 12:52:39 +03:00
b499c65ac8 Added common funcs. Added config example 2024-07-01 12:25:52 +03:00
0b61461039 Added donwloading tracks and files 2024-07-01 11:28:18 +03:00
2279f572cb Added Folder and filename generation 2024-06-28 15:17:14 +03:00
ee703e6fde Added exteneded mix parsing 2024-06-28 14:06:59 +03:00
16 changed files with 917 additions and 1175 deletions

3
.gitignore vendored
View File

@@ -1,5 +1,4 @@
login.json
release.json
config.json
cookies.txt
mcat_dl*
.vscode/*

View File

@@ -1,4 +1,9 @@
srcs += main.cpp
srcs += curl_dl.cpp
srcs += monstercat_dl.cpp
srcs += common.cpp
all:
g++ main.cpp -o mcat_dl
g++ ${srcs} -lcurl -o mcat_dl
.PHONY: all

54
README
View File

@@ -1,54 +0,0 @@
--- Usage ---
There must be a file named "login.json" in the working directory with your monstercat credentials.
See example_login.json for the template.
--- JSON Library Source ---
https://github.com/nlohmann/json
Release - 3.10.5
Commit - 4f8fba14066156b73f1189a2b8bd568bde5284c5
--- id3edit ---
https://github.com/rstemmer/id3edit
For MP3 tagging
id3edit
--set-name "Title"
--set-album "Album"
--set-artist "Artist"
--set-track "Track Number"
--set-artwork "/path/to/cover"
file.mp3
--- metaflac ---
https://xiph.org/flac/download.html
For FLAC tagging
metaflac
// Common
--preserve-modtime
--no-utf8-convert
// First Step - Remove Tags
--remove-tag=TITLE
--remove-tag=ARTIST
--remove-tag=ALBUM
--remove-tag=TRACKNUMBER
// Second Step - Remove Pictures
--remove --block-type=PICTURE
// Third Step - Add
--import-picture-from=3|image/jpeg|||"/path/to/cover"
"--set-tag=TITLE=..."
"--set-tag=ARTIST=..."
"--set-tag=ALBUM=..."
"--set-tag=TRACKNUMBER=..."
--dont-use-padding
file.flac
--- imagemagick ---
https://imagemagick.org/script/download.php
For Image Resizing
magick
Cover
-resize 750x750
Cover_small.jpg

109
README.md Normal file
View File

@@ -0,0 +1,109 @@
# Prerequisites
### Ubuntu
`apt install build-essential libcurl4-gnutls-dev imagemagick eyeD3 flac`
### Other OS
* make
* g++
* libcurl
* imagemagick (convert executable)
* eyeD3 executable
* metaflac executable
# Usage
There must be a file named "config.json" in the working directory with your monstercat credentials and other configurations.
See config_example.json for the template.
Invoke `make` to compile. Requires libcurl.
Login via browser and export `cid` cookie to cookies.txt (Netscape format)
Invoke `mcat_dl` with catalog ID as extra arguments to download.
# Download File Structure
### Release Folder
**download_path/Type/YYYY-MM-DD - CatalogID - Artist - Title**
* download_path - Set in config.json
* Type - Album/EP/Single
* YYYY-MM-DD - Release Date in ISO format for easy sorting
* Artist/Title - removed featuring artists from "Artist" and added to "Title"
### Cover
* Cover(.jpg/.png) - Full resolution from Monstercat
* Cover_small.jpg - 750x750 used to tag files
### MP3 and FLAC folders
Separate folders if more than 1 track otherwise put in main folder.
### Extended folder
This is where extended mixes are put into. They are in their original format.
### File Naming
* Title.(mp3/flac) - Used when only 1 track
* Number - Title.(mp3/flac) - Used when track artist **matches** release artist
* Number - Artist - Title.(mp3/flac) - Used when track artist **does not match** release artist
# JSON Library Source
https://github.com/nlohmann/json
* Release - 3.10.5
* Commit - 4f8fba14066156b73f1189a2b8bd568bde5284c5
# eyeD3
https://eyed3.readthedocs.io/en/latest/
* For MP3 tagging
```
// Common
--encoding utf8
--preserve-file-times
// First Step - Remove Images
--remove-all-images
// Second Step - Change what is needed
--add-image "...":FRONT_COVER
--artist "..."
--album "..."
--title "..."
--track ...
```
# metaflac
https://xiph.org/flac/download.html
* For FLAC tagging
```
metaflac
// Common
--preserve-modtime
--no-utf8-convert
// First Step - Remove Pictures
--remove --block-type=PICTURE
// Second Step - Remove Tags
--remove-tag=TITLE
--remove-tag=ARTIST
--remove-tag=ALBUM
--remove-tag=TRACKNUMBER
// Third Step - Add
"--import-picture-from=3|image/jpeg|||/path/to/cover"
"--set-tag=TITLE=..."
"--set-tag=ARTIST=..."
"--set-tag=ALBUM=..."
"--set-tag=TRACKNUMBER=..."
--dont-use-padding
file.flac
```
# imagemagick
https://imagemagick.org/script/download.php
* For Image Resizing
```
convert
Cover
-resize 750x750
Cover_small.jpg
```

0
TODO
View File

1
TODO.md Normal file
View File

@@ -0,0 +1 @@
* Add API Rate Limit

203
common.cpp Normal file
View File

@@ -0,0 +1,203 @@
#include "common.h"
#include <fstream>
#include <set>
#include <stdlib.h>
#include <sys/stat.h>
using std::endl;
using std::ifstream;
using std::istream;
using std::ostream;
using std::set;
using std::string;
using std::to_string;
using nlohmann::json;
bool read_config(json& cfg, ostream* log)
{
ifstream cfg_file("config.json");
if (!cfg_file.is_open())
{
if (nullptr != log)
{
*log << "Could not open config.json" << endl;
}
return false;
}
// Parse with comments
try
{
cfg = json::parse(cfg_file, nullptr, true, true);
}
catch (const std::exception &e)
{
if (nullptr != log)
{
*log << e.what() << endl;
}
return false;
}
return true;
}
bool path_exists(const string& path)
{
struct stat info;
int error;
error = stat(path.c_str(), &info);
if (0 != error)
{
return false;
}
if (S_ISDIR(info.st_mode))
{
return true;
}
return false;
}
string build_fname(const string& main_path, const string& folder, const string& fname)
{
string path_to_file;
path_to_file = main_path;
if (!path_to_file.empty() && path_to_file[path_to_file.size() - 1] != '/')
{
path_to_file += '/';
}
path_to_file += folder;
if (!path_to_file.empty() && path_to_file[path_to_file.size() - 1] != '/')
{
path_to_file += '/';
}
path_to_file += fname;
return path_to_file;
}
string i_to_str(int i, int size, char fill)
{
string str;
str = to_string(i);
str.insert(str.begin(), size - str.size(), fill);
return str;
}
void json_extract(const json& j, const string& key, string& out)
{
if (j.contains(key) && j[key].is_string())
{
out = j[key];
}
}
void json_extract(const json& j, const string& key, uint16_t& out)
{
if (j.contains(key) && j[key].is_number_unsigned())
{
out = j[key];
}
}
void json_extract(const json& j, const string& key, bool& out)
{
if (j.contains(key) && j[key].is_boolean())
{
out = j[key];
}
}
bool ensure_folder(const string& main_path, const string& folder)
{
string full_path;
int error;
if (!path_exists(main_path))
{
error = mkdir(main_path.c_str(), 0775);
if (0 != error)
{
return false;
}
}
full_path = build_fname(main_path, folder, "");
if (path_exists(full_path))
{
return true;
}
error = mkdir(full_path.c_str(), 0775);
if (0 != error)
{
return false;
}
return true;
}
string trim_whitespace(const string& s)
{
const string whitespace = " \t\r\n\f\v";
string result(s);
size_t pos;
pos = result.find_first_not_of(whitespace);
result.erase(0, pos);
pos = result.find_last_not_of(whitespace);
result.erase(pos + 1);
return result;
}
string clean_filename(const string& s)
{
string result = s;
set<char> illegal;
illegal.insert('/');
illegal.insert('\\');
illegal.insert('<');
illegal.insert('>');
illegal.insert(':');
illegal.insert('"');
illegal.insert('|');
illegal.insert('?');
illegal.insert('*');
for (char& c : result)
{
if (0 != illegal.count(c))
{
c = ';';
}
}
return result;
}
bool exec_cmd(const string& cmd)
{
int error;
error = system(cmd.c_str());
if (0 != error)
{
return false;
}
return true;
}

24
common.h Normal file
View File

@@ -0,0 +1,24 @@
#ifndef COMMON_H_
#define COMMON_H_
#include <iostream>
#include <set>
#include <string>
#include "json.hpp"
bool read_config(nlohmann::json& cfg, std::ostream* log = &std::cout);
bool path_exists(const std::string& path);
std::string build_fname(const std::string& main_path, const std::string& folder,
const std::string& fname);
std::string i_to_str(int i, int size, char fill = '0');
void json_extract(const nlohmann::json& j, const std::string& key,
std::string& out);
void json_extract(const nlohmann::json& j, const std::string& key,
uint16_t& out);
void json_extract(const nlohmann::json& j, const std::string& key, bool& out);
bool ensure_folder(const std::string& main_path, const std::string& folder);
std::string trim_whitespace(const std::string& s);
std::string clean_filename(const std::string& s);
bool exec_cmd(const std::string& cmd);
#endif // COMMON_H_

8
config_example.json Normal file
View File

@@ -0,0 +1,8 @@
{
"Email": "user@example.com",
"Pass": "password",
"download_path": "/path/to/monstercat",
"convert_exec": "convert",
"eyed3_exec": "eyeD3",
"metaflac_exec": "metaflac"
}

View File

@@ -185,13 +185,13 @@ bool CURL_DL::download(const string& url, ostream* out,
curl_easy_reset(m_handle);
// Enable cookie engine
curl_easy_setopt(m_handle, CURLOPT_COOKIEFILE, "");
curl_easy_setopt(m_handle, CURLOPT_COOKIEFILE, "cookies.txt");
// Enable error messages
curl_easy_setopt(m_handle, CURLOPT_ERRORBUFFER, m_error);
// Set User-Agent
curl_easy_setopt(m_handle, CURLOPT_USERAGENT, "Internedko Archiver");
curl_easy_setopt(m_handle, CURLOPT_USERAGENT, "Internedko Downloader");
if ((nullptr == params) || (params->count("no-redir") == 0))
{

View File

@@ -1,7 +0,0 @@
{
"comment": "This file will attempt to download MCLP001, MCLP001-X, MCLP002, ... MCLP010-X",
"prefix": "MCLP",
"suffix_try": "-X",
"start": 1,
"end": 10
}

View File

@@ -1,10 +0,0 @@
{
"comment": "This file will attempt to download MCS1091, MCS1195, MCS1425, MCS1426",
"releases":
[
"MCS1091",
"MCS1195",
"MCS1425",
"MCS1426"
]
}

View File

@@ -1,4 +0,0 @@
{
"Email": "USER",
"Password": "PASS"
}

1304
main.cpp

File diff suppressed because it is too large Load Diff

View File

@@ -1,44 +1,21 @@
#include "monstercat_dl.h"
#include "common.h"
#include "curl_dl.h"
#include <algorithm>
#include <fstream>
#include <sstream>
using std::endl;
using std::map;
using std::ofstream;
using std::sort;
using std::string;
using std::stringstream;
using std::to_string;
using nlohmann::json;
static bool json_extract(const json& j, const string& key, string& value)
{
bool result = false;
if (j.contains(key) && j[key].is_string())
{
value = j[key];
result = true;
}
return result;
}
static string trim_whitespace(const string& s)
{
const string whitespace = " \t\r\n\f\v";
string result(s);
size_t pos;
pos = result.find_first_not_of(whitespace);
result.erase(0, pos);
pos = result.find_last_not_of(whitespace);
result.erase(pos + 1);
return result;
}
string Monstercat_DL::calc_proper_artist(const string& artist_raw)
{
size_t pos;
@@ -78,10 +55,12 @@ string Monstercat_DL::calc_proper_title(const string& artist_raw,
result += trim_whitespace(version_raw);
result += ")";
}
return result;
}
Monstercat_DL::Monstercat_DL()
: log(&std::cout), m_base_url("https://player.monstercat.app/api/"),
: m_log(&std::cout), m_base_url("https://player.monstercat.app/api/"),
m_is_logged_in(false)
{}
@@ -106,10 +85,11 @@ bool Monstercat_DL::login(const string& user, const string& pass)
ok = curl.post_json(url, data, &out);
if (!ok)
{
if (nullptr != log)
if (nullptr != m_log)
{
*log << "Could not post json" << endl;
*log << "CURL:" << curl.get_error() << endl;
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not post json" << endl;
*m_log << "CURL:" << curl.get_error() << endl;
}
return false;
}
@@ -146,10 +126,11 @@ bool Monstercat_DL::logout()
ok = curl.post_json(url, data, nullptr);
if (!ok)
{
if (nullptr != log)
if (nullptr != m_log)
{
*log << "Could not post json" << endl;
*log << "CURL:" << curl.get_error() << endl;
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not post json" << endl;
*m_log << "CURL:" << curl.get_error() << endl;
}
return false;
}
@@ -172,10 +153,11 @@ json Monstercat_DL::get_release_json(const string& catalog_id)
ok = curl.download(url, &out);
if (!ok)
{
if (nullptr != log)
if (nullptr != m_log)
{
*log << "Could not download json" << endl;
*log << "CURL:" << curl.get_error() << endl;
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not download json" << endl;
*m_log << "CURL:" << curl.get_error() << endl;
}
return result;
}
@@ -199,10 +181,11 @@ json Monstercat_DL::get_browse_json(const string& release_id)
ok = curl.download(url, &out);
if (!ok)
{
if (nullptr != log)
if (nullptr != m_log)
{
*log << "Could not download json" << endl;
*log << "CURL:" << curl.get_error() << endl;
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not download json" << endl;
*m_log << "CURL:" << curl.get_error() << endl;
}
return result;
}
@@ -231,6 +214,7 @@ Release Monstercat_DL::parse_release_json(const json& release_json)
}
// Parse Release
json_extract(release_object, "CatalogId", result.catalog_id);
json_extract(release_object, "Id", result.id);
json_extract(release_object, "Type", result.type);
json_extract(release_object, "ReleaseDate", result.release_date);
@@ -278,9 +262,173 @@ Release Monstercat_DL::parse_release_json(const json& release_json)
result.tracks.push_back(temp_track);
}
// Sort tracks by number
sort(result.tracks.begin(), result.tracks.end(), [](const Track& t1, const Track& t2)
{
return t1.number < t2.number;
});
return result;
}
void Monstercat_DL::add_extended_mixes(Release& release, const json& browse_json)
{
int track_num;
json data_obj;
json file_obj;
Track *track;
string mime_type;
if (browse_json.contains("Data") && browse_json["Data"].is_array())
{
data_obj = browse_json["Data"];
}
for (json& track_json : data_obj)
{
// Get the track number
track_num = 0;
if (track_json.contains("TrackNumber") && track_json["TrackNumber"].is_number_integer())
{
track_num = track_json["TrackNumber"];
}
// File must exist and be an object
if (!track_json.contains("File") || !track_json["File"].is_object())
{
continue;
}
// Find track
track = nullptr;
for (int i = 0; i < release.tracks.size(); ++i)
{
if (release.tracks[i].number == track_num)
{
track = &(release.tracks[i]);
break;
}
}
if (nullptr == track)
{
if (nullptr != m_log)
{
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not find track number " << track_num << " for catalog id " << release.catalog_id << endl;
}
return;
}
file_obj = track_json["File"];
json_extract(file_obj, "Id", track->extended_mix_file_id);
json_extract(file_obj, "MimeType", mime_type);
if ("audio/wav" == mime_type)
{
track->extended_mix_extension = ".wav";
}
else
{
if (nullptr != m_log)
{
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Unknown MIME type for catalog id " << release.catalog_id << " - " << mime_type << endl;
}
}
return;
}
}
string Monstercat_DL::calc_release_folder(const Release& release, const string& main_folder)
{
string result;
string folder_name;
ensure_folder(main_folder, release.type);
folder_name = release.release_date;
folder_name += " - ";
folder_name += clean_filename(release.catalog_id);
folder_name += " - ";
folder_name += clean_filename(release.artist);
folder_name += " - ";
folder_name += clean_filename(release.title);
result = build_fname(main_folder, release.type, folder_name);
ensure_folder(result, "");
return result;
}
string Monstercat_DL::calc_track_filename(const Release& release, int track_num)
{
string result;
bool use_number;
bool use_artist;
const Track *track = nullptr;
// Find the track
for (int i = 0; i < release.tracks.size(); ++i)
{
if (release.tracks[i].number == track_num)
{
track = &(release.tracks[i]);
break;
}
}
if (nullptr == track)
{
if (nullptr != m_log)
{
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Cannot find track number " << track_num << " in catalog id " << release.catalog_id << endl;
}
return "";
}
// Determine whether number is needed
if (1 == release.tracks.size())
{
use_number = false;
}
else
{
use_number = true;
}
// Determine whether artist is needed
if (track->artist == release.artist)
{
use_artist = false;
}
else
{
use_artist = true;
}
// Build filename
result = "";
if (use_number)
{
// Always use 2 digits
result += i_to_str(track->number, 2, '0');
result += " - ";
}
if (use_artist)
{
result += track->artist;
result += " - ";
}
result += track->title;
return clean_filename(result);
}
bool Monstercat_DL::download_cover(const string& catalog_id, const string& path)
{
CURL_DL& curl = CURL_DL::get_handle();
@@ -289,7 +437,7 @@ bool Monstercat_DL::download_cover(const string& catalog_id, const string& path)
stringstream out;
ofstream out_file;
map<string, string> out_headers;
string filename;
string filepath;
url = "https://www.monstercat.com/release/";
url += catalog_id;
@@ -298,48 +446,52 @@ bool Monstercat_DL::download_cover(const string& catalog_id, const string& path)
ok = curl.download(url, &out, nullptr, &out_headers);
if (!ok)
{
if (nullptr != log)
if (nullptr != m_log)
{
*log << "Could not download image" << endl;
*log << "CURL:" << curl.get_error() << endl;
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not download image" << endl;
*m_log << "CURL:" << curl.get_error() << endl;
}
return false;
}
filename = path;
filepath = build_fname(path, "", "Cover");
if (0 == out_headers.count("content-type"))
{
if (nullptr != log)
if (nullptr != m_log)
{
*log << "No content-type" << endl;
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "No content-type" << endl;
}
}
else
{
if (out_headers["content-type"] == "image/jpeg")
{
filename += ".jpg";
filepath += ".jpg";
}
else if (out_headers["content-type"] == "image/png")
{
filename += ".png";
filepath += ".png";
}
else
{
if (nullptr != log)
if (nullptr != m_log)
{
*log << "Unknown Content-Type for cover - " << out_headers["content-type"] << endl;
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Unknown Content-Type for cover - " << out_headers["content-type"] << endl;
}
}
}
out_file.open(filename, std::ios::binary);
out_file.open(filepath, std::ios::binary);
if (!out_file.is_open())
{
if (nullptr != log)
if (nullptr != m_log)
{
*log << "Could not open file for write" << endl;
*log << filename << endl;
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not open file for write" << endl;
*m_log << filepath << endl;
}
return false;
}
@@ -351,27 +503,97 @@ bool Monstercat_DL::download_cover(const string& catalog_id, const string& path)
}
bool Monstercat_DL::download_track(const string& release_id,
const string& track_id, const string& path)
const string& track_id, const string& filepath, bool is_mp3)
{
CURL_DL& curl = CURL_DL::get_handle();
bool ok;
string url;
stringstream out_data;
ofstream out_file;
url = m_base_url;
url += "release/";
url += release_id;
url += "/track-download/";
url += track_id;
if (is_mp3)
{
url += "?format=mp3_320";
}
else
{
url += "?format=flac";
}
return false;
ok = curl.download(url, &out_data);
if (!ok)
{
if (nullptr != m_log)
{
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not download track" << endl;
*m_log << "CURL:" << curl.get_error() << endl;
}
return false;
}
out_file.open(filepath, std::ios::binary);
if (!out_file.is_open())
{
if (nullptr != m_log)
{
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not open file for write" << endl;
*m_log << filepath << endl;
}
return false;
}
out_file << out_data.rdbuf();
out_file.close();
return true;
}
bool Monstercat_DL::download_file(const string& file_id, const string& path)
bool Monstercat_DL::download_file(const string& file_id, const string& filepath)
{
CURL_DL& curl = CURL_DL::get_handle();
bool ok;
string url;
stringstream out_data;
ofstream out_file;
url = m_base_url;
url += "file/";
url += file_id;
url += "/open?download=true";
return false;
ok = curl.download(url, &out_data);
if (!ok)
{
if (nullptr != m_log)
{
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not download track" << endl;
*m_log << "CURL:" << curl.get_error() << endl;
}
return false;
}
out_file.open(filepath, std::ios::binary);
if (!out_file.is_open())
{
if (nullptr != m_log)
{
*m_log << __PRETTY_FUNCTION__ << endl;
*m_log << "Could not open file for write" << endl;
*m_log << filepath << endl;
}
return false;
}
out_file << out_data.rdbuf();
out_file.close();
return true;
}

View File

@@ -6,6 +6,9 @@
#include <vector>
#include "json.hpp"
constexpr bool FORMAT_MP3 = true;
constexpr bool FORMAT_FLAC = false;
struct Track
{
public:
@@ -27,12 +30,13 @@ struct Release
std::vector<Track> tracks;
std::string id;
std::string catalog_id;
};
class Monstercat_DL
{
private:
std::ostream *log;
std::ostream *m_log;
std::string m_base_url;
@@ -53,11 +57,15 @@ public:
nlohmann::json get_browse_json(const std::string& release_id);
Release parse_release_json(const nlohmann::json& release_json);
void add_extended_mixes(Release& release, const nlohmann::json& browse_json);
std::string calc_release_folder(const Release& release, const std::string& main_folder);
std::string calc_track_filename(const Release& release, int track_num);
bool download_cover(const std::string& catalog_id, const std::string& path);
bool download_track(const std::string& release_id,
const std::string& track_id, const std::string& path);
bool download_file(const std::string& file_id, const std::string& path);
const std::string& track_id, const std::string& filepath, bool is_mp3);
bool download_file(const std::string& file_id, const std::string& filepath);
};