 Lihat di AI Google
|
 Coba notebook Colab
|
 Lihat notebook di GitHub
|
Dalam tutorial ini, Anda akan mempelajari contoh ekstraksi data terstruktur, menggunakan Gemini API untuk mengekstrak daftar karakter, hubungan, benda, dan tempat dari sebuah cerita.
Penyiapan
pip install -U -q google-generativeai
import pathlib
import textwrap
import google.generativeai as genai
from IPython.display import display
from IPython.display import Markdown
from google.api_core import retry
def to_markdown(text):
text = text.replace('•', ' *')
return Markdown(textwrap.indent(text, '> ', predicate=lambda _: True))
Setelah Anda memiliki kunci API, teruskan ke SDK. Anda dapat melakukannya dengan dua cara:
- Masukkan kunci di variabel lingkungan
GOOGLE_API_KEY(SDK akan otomatis mengambilnya dari sana). - Teruskan kunci ke
genai.configure(api_key=...)
genai.configure(api_key=GOOGLE_API_KEY)
Contoh tugas
Untuk tutorial ini, Anda akan mengekstrak entity dari cerita natural language. Sebagai seorang contohnya, di bawah ini adalah cerita yang ditulis oleh Gemini.
new_story = False
if new_story:
model = genai.GenerativeModel(model_name='models/gemini-1.5-pro-latest')
response = model.generate_content("""
Write a long story about a girl with magic backpack, her family, and at
least one other charater. Make sure everyone has names. Don't forget to
describe the contents of the backpack, and where everyone and everything
starts and ends up.""", request_options={'retry': retry.Retry()})
story = response.text
print(response.candidates[0].citation_metadata)
else:
story = """In the quaint town of Willow Creek, nestled amidst rolling hills and whispering willows, resided a young girl named Anya. As she stepped out of the creaky wooden door of her modest cottage, her heart skipped a beat with excitement and anticipation. Today was her first day of school, and she couldn't wait to show off her prized possession - a magical backpack.\n\nHanded down to her from her grandmother, the backpack was no ordinary satchel. Its soft, emerald-green fabric shimmered with an ethereal glow, and its leather straps held secrets that only Anya knew. Within its capacious interior lay an enchanted world, filled with wonders that would ignite her imagination and change her life forever.\n\nAnya's parents, kind-hearted Elise and wise-bearded Edward, bid her farewell with warm embraces. "Remember, my dear," whispered her mother, "use your magic wisely and for good." Her father added, "Always seek knowledge, and let the backpack be your trusted companion."\n\nWith a skip in her step, Anya set off towards the town's only schoolhouse. On her way, she passed her best friend, Samuel, a curious and adventurous boy with a mischievous grin. "Hey, Anya," he called out. "Can I see your backpack?"\n\nAnya hesitated for a moment before unzipping the flap and revealing its contents. Samuel's eyes widened in amazement as he peered inside. There, nestled amidst pencils and notebooks, were a shimmering sword, a book of ancient spells, a tiny compass that always pointed north, and a magical key that could open any lock.\n\nTogether, they marveled at the backpack's wonders, promising to keep its secrets safe. As they approached the schoolhouse, Anya noticed a group of older children huddled together, their faces etched with fear. Curiosity getting the better of her, she cautiously approached.\n\n"What's wrong?" she asked.\n\nA tall, lanky boy stepped forward. "There's a monster in the forest," he stammered. "It's been terrorizing the town, attacking animals and even people."\n\nAnya's heart sank. The town of Willow Creek was small and peaceful, and the thought of a monster brought a shiver down her spine. She knew she had to do something to protect her family and friends.\n\nWithout a moment's hesitation, Anya opened her backpack and retrieved the shimmering sword. With a determined gleam in her eye, she turned to her terrified peers. "Don't worry," she said, her voice steady. "I'll take care of it."\n\nWith Samuel close behind her, Anya ventured into the shadowy depths of the forest. The trees seemed to whisper secrets as she passed, and the undergrowth rustled with unseen creatures. As they walked deeper into the forest, the air grew heavy and the ground beneath their feet trembled.\n\nSuddenly, they came to a clearing, and there before their eyes was the monster - a massive beast with sharp teeth, glowing red eyes, and claws that could crush a human with ease. The creature roared, a thunderous sound that shook the forest to its core.\n\nFear surged through Anya, but she refused to let it consume her. She drew the sword from its sheath and charged towards the monster. The blade shimmered in the sunlight, and as it struck the beast's hide, a blinding light erupted, enveloping everything in its radiance.\n\nWhen the light faded, the monster was gone, and in its place was a pile of shattered crystals. Anya had defeated the creature with the magic of her backpack, proving that even the smallest of objects could hold the greatest of powers.\n\nAs she and Samuel returned to the town, they were greeted as heroes. The people of Willow Creek rejoiced, and the legend of Anya, the girl with the magic backpack, was passed down through generations. And so, Anya continued her adventures, using the backpack's wonders to make the world a better place, one magical step at a time."""
to_markdown(story)
Di kota kuno Willow Creek, terletak di tengah-tengah perbukitan dan pohon dedalu yang berbisik, tinggal seorang gadis muda bernama Anya. Saat dia melangkah keluar dari pintu kayu berderit di pondoknya yang sederhana, jantungnya berdebar kencang dengan kegembiraan dan antisipasi. Hari ini adalah hari pertama sekolahnya, dan dia tidak sabar untuk memamerkan harta karunnya yang berharga, yaitu ransel ajaib.
Dibawakan kepadanya dari neneknya, ransel itu bukanlah tas biasa. Kainnya yang lembut berwarna hijau zamrud berkilau dengan kilau lembut, dan tali kulitnya menyimpan rahasia yang hanya diketahui Anya. Di dalam interiornya yang luas terdapat sebuah dunia yang penuh keajaiban, penuh dengan keajaiban yang akan memicu imajinasi dan mengubah hidupnya selamanya.
Orang tua Anya, Elise yang baik hati dan Edward yang berjanggut bijaksana, mengucapkan selamat tinggal dengan pelukan hangat kepada Anya. "Ingat, sayangku," bisik ibunya, "gunakan sihirmu dengan bijak dan untuk kebaikan." Ayahnya menambahkan, "Selalu berusahalah untuk memperoleh ilmu, dan biarkan ransel menjadi teman yang tepercaya."
Dengan melewati langkahnya, Anya berangkat menuju satu-satunya gedung sekolah di kota itu. Dalam perjalanannya, dia berpapasan dengan sahabatnya, Samuel, seorang anak laki-laki yang ingin tahu dan suka berpetualang dengan senyum nakal. "Hai, Anya," teriaknya. "Apa aku bisa melihat ranselmu?"
Anya ragu-ragu sejenak sebelum membuka ritsleting sayap dan memperlihatkan isinya. Mata Samuel melebar kagum saat dia mengintip ke dalam. Di situ, di antara pensil dan buku catatan, terdapat pedang yang berkilauan, buku mantra kuno, kompas kecil yang selalu mengarah ke utara, dan kunci ajaib yang dapat membuka gembok apa pun.
Bersama-sama, mereka mengagumi keajaiban ransel, menjanjikan untuk menjaga rahasianya tetap aman. Saat mereka mendekati gedung sekolah, Anya melihat sekelompok anak yang lebih tua berkumpul bersama, wajah mereka terukir ketakutan. Karena rasa ingin tahunya semakin kuat, ia mendekatinya dengan hati-hati.
"Apa yang salah?" tanyanya.
Seorang anak laki-laki tinggi dan kurus melangkah ke depan. "Ada monster di hutan." dia terbata. "Saat ini telah meneror kota, menyerang hewan dan bahkan manusia."
Hati Anya tenggelam. Kota Willow Creek adalah kota yang kecil dan damai, dan pikiran tentang monster membuat tulang belakangnya menggigil. Dia tahu bahwa dia harus melakukan sesuatu untuk melindungi keluarga dan teman-temannya.
Tanpa ragu, Anya membuka ranselnya dan mengambil pedang yang berkilauan itu. Dengan mata berbinar-binar penuh semangat, dia menoleh kepada teman-temannya yang ketakutan. "Jangan khawatir." katanya, suaranya stabil. "Aku yang akan mengurusnya."
Dengan Samuel berada di belakangnya, Anya bertualang ke kedalaman hutan yang gelap. Pepohonan tampak membisikkan rahasia saat ia lewat, dan semak-semak itu penuh dengan makhluk yang tak terlihat. Saat mereka berjalan semakin dalam ke dalam hutan, udara menjadi semakin lebat dan tanah di bawah kaki mereka bergetar.
Tiba-tiba, mereka sampai di tempat terbuka, dan di depan mata mereka ada monster itu - binatang buas besar dengan gigi tajam, mata merah bersinar, dan cakar yang dapat menaklukkan manusia dengan mudah. Makhluk itu meraung, suara menggelegar yang menggoyangkan hutan hingga ke intinya.
Rasa takut meledak melalui Anya, tetapi dia menolak membiarkan Anya memakannya. Dia menarik pedang dari sarungnya dan menyerang monster itu. Pisau itu bercahaya di bawah sinar matahari, dan saat menghantam kulit binatang buas itu, cahaya yang menyadarkan meledak, menyelimuti segala sesuatu dalam pancarannya.
Ketika cahaya memudar, monster itu menghilang, dan sebagai gantinya adalah tumpukan kristal yang hancur. Anya telah mengalahkan makhluk itu dengan keajaiban ranselnya, membuktikan bahwa benda terkecil pun dapat menyimpan kekuatan terbesar.
Saat ia dan Samuel kembali ke kota, mereka disambut sebagai pahlawan. Orang-orang Willow Creek bersuka cita, dan legenda Anya, gadis dengan ransel ajaib, diwariskan turun-temurun. Lalu, Anya melanjutkan petualangannya, menggunakan keajaiban ransel untuk membuat dunia menjadi tempat yang lebih baik, selangkah demi selangkah.
Menggunakan Natural language
Model bahasa besar adalah alat multitasking yang canggih. Sering kali Anda cukup meminta Gemini untuk melakukan apa pun yang Anda inginkan, dan Gemini akan baik-baik saja.
Gemini API tidak memiliki mode JSON, jadi ada beberapa hal yang harus diperhatikan saat membuat struktur data dengan cara ini:
- Terkadang penguraian gagal.
- Skema tidak dapat diterapkan secara ketat.
Anda akan menyelesaikan masalah tersebut di bagian berikutnya. Pertama, coba perintah natural language sederhana dengan skema yang ditulis sebagai teks. Ini belum dioptimalkan:
model = genai.GenerativeModel(
model_name='models/gemini-1.5-pro-latest')
response = model.generate_content(
textwrap.dedent("""\
Please return JSON describing the the people, places, things and relationships from this story using the following schema:
{"people": list[PERSON], "places":list[PLACE], "things":list[THING], "relationships": list[RELATIONSHIP]}
PERSON = {"name": str, "description": str, "start_place_name": str, "end_place_name": str}
PLACE = {"name": str, "description": str}
THING = {"name": str, "description": str, "start_place_name": str, "end_place_name": str}
RELATIONSHIP = {"person_1_name": str, "person_2_name": str, "relationship": str}
All fields are required.
Important: Only return a single piece of valid JSON text.
Here is the story:
""") + story,
generation_config={'response_mime_type':'application/json'}
)
response.text
'{"people": [\n {\n "name": "Anya",\n "description": "A young girl who lives in the town of Willow Creek with her parents, Elise and Edward. She possesses a magical backpack that was handed down to her from her grandmother.",\n "start_place_name": "Willow Creek",\n "end_place_name": "Willow Creek"\n },\n {\n "name": "Elise",\n "description": "Anya\'s kind-hearted mother",\n "start_place_name": "Willow Creek",\n "end_place_name": "Willow Creek"\n },\n {\n "name": "Edward",\n "description": "Anya\'s wise-bearded father",\n "start_place_name": "Willow Creek",\n "end_place_name": "Willow Creek"\n },\n {\n "name": "Samuel",\n "description": "Anya\'s best friend, a curious and adventurous boy with a mischievous grin.",\n "start_place_name": "Willow Creek",\n "end_place_name": "Willow Creek"\n },\n {\n "name": "Monster",\n "description": "A massive beast with sharp teeth, glowing red eyes, and claws that could crush a human with ease.",\n "start_place_name": "Forest",\n "end_place_name": "Forest"\n }\n], "places": [\n {\n "name": "Willow Creek",\n "description": "A quaint town nestled amidst rolling hills and whispering willows."\n },\n {\n "name": "Forest",\n "description": "A shadowy place with rustling undergrowth and whispering trees."\n },\n {\n "name": "Schoolhouse",\n "description": "The only school in the town of Willow Creek."\n },\n {\n "name": "Anya\'s home",\n "description": "A modest cottage with a creaky wooden door."\n }\n], "things": [\n {\n "name": "Magic backpack",\n "description": "A magical backpack that was handed down to Anya from her grandmother. Its soft, emerald-green fabric shimmered with an ethereal glow, and its leather straps held secrets that only Anya knew.",\n "start_place_name": "Anya\'s home",\n "end_place_name": "Forest"\n },\n {\n "name": "Shimmering sword",\n "description": "A sword that shimmered in the sunlight and could strike with blinding light.",\n "start_place_name": "Magic backpack",\n "end_place_name": "Forest"\n },\n {\n "name": "Book of ancient spells",\n "description": "A book that contained ancient spells.",\n "start_place_name": "Magic backpack",\n "end_place_name": "Forest"\n },\n {\n "name": "Tiny compass",\n "description": "A compass that always pointed north.",\n "start_place_name": "Magic backpack",\n "end_place_name": "Forest"\n },\n {\n "name": "Magical key",\n "description": "A key that could open any lock.",\n "start_place_name": "Magic backpack",\n "end_place_name": "Forest"\n },\n {\n "name": "Shattered crystals",\n "description": "The remains of the monster after it was defeated by Anya\'s magic backpack.",\n "start_place_name": "Forest",\n "end_place_name": "Forest"\n }\n], "relationships": [\n {\n "person_1_name": "Anya",\n "person_2_name": "Elise",\n "relationship": "mother-daughter"\n },\n {\n "person_1_name": "Anya",\n "person_2_name": "Edward",\n "relationship": "father-daughter"\n },\n {\n "person_1_name": "Anya",\n "person_2_name": "Samuel",\n "relationship": "best friends"\n }\n]}'
Perintah ini menampilkan string json. Coba uraikan:
import json
print(json.dumps(json.loads(response.text), indent=4))
{
"people": [
{
"name": "Anya",
"description": "A young girl who lives in the town of Willow Creek with her parents, Elise and Edward. She possesses a magical backpack that was handed down to her from her grandmother.",
"start_place_name": "Willow Creek",
"end_place_name": "Willow Creek"
},
{
"name": "Elise",
"description": "Anya's kind-hearted mother",
"start_place_name": "Willow Creek",
"end_place_name": "Willow Creek"
},
{
"name": "Edward",
"description": "Anya's wise-bearded father",
"start_place_name": "Willow Creek",
"end_place_name": "Willow Creek"
},
{
"name": "Samuel",
"description": "Anya's best friend, a curious and adventurous boy with a mischievous grin.",
"start_place_name": "Willow Creek",
"end_place_name": "Willow Creek"
},
{
"name": "Monster",
"description": "A massive beast with sharp teeth, glowing red eyes, and claws that could crush a human with ease.",
"start_place_name": "Forest",
"end_place_name": "Forest"
}
],
"places": [
{
"name": "Willow Creek",
"description": "A quaint town nestled amidst rolling hills and whispering willows."
},
{
"name": "Forest",
"description": "A shadowy place with rustling undergrowth and whispering trees."
},
{
"name": "Schoolhouse",
"description": "The only school in the town of Willow Creek."
},
{
"name": "Anya's home",
"description": "A modest cottage with a creaky wooden door."
}
],
"things": [
{
"name": "Magic backpack",
"description": "A magical backpack that was handed down to Anya from her grandmother. Its soft, emerald-green fabric shimmered with an ethereal glow, and its leather straps held secrets that only Anya knew.",
"start_place_name": "Anya's home",
"end_place_name": "Forest"
},
{
"name": "Shimmering sword",
"description": "A sword that shimmered in the sunlight and could strike with blinding light.",
"start_place_name": "Magic backpack",
"end_place_name": "Forest"
},
{
"name": "Book of ancient spells",
"description": "A book that contained ancient spells.",
"start_place_name": "Magic backpack",
"end_place_name": "Forest"
},
{
"name": "Tiny compass",
"description": "A compass that always pointed north.",
"start_place_name": "Magic backpack",
"end_place_name": "Forest"
},
{
"name": "Magical key",
"description": "A key that could open any lock.",
"start_place_name": "Magic backpack",
"end_place_name": "Forest"
},
{
"name": "Shattered crystals",
"description": "The remains of the monster after it was defeated by Anya's magic backpack.",
"start_place_name": "Forest",
"end_place_name": "Forest"
}
],
"relationships": [
{
"person_1_name": "Anya",
"person_2_name": "Elise",
"relationship": "mother-daughter"
},
{
"person_1_name": "Anya",
"person_2_name": "Edward",
"relationship": "father-daughter"
},
{
"person_1_name": "Anya",
"person_2_name": "Samuel",
"relationship": "best friends"
}
]
}
Cara ini relatif sederhana dan sering kali berhasil, tetapi Anda dapat membuatnya lebih ketat/kokoh dengan menentukan skema menggunakan fitur panggilan fungsi API.
Gunakan panggilan fungsi
Jika Anda belum mempelajari tutorial Dasar-dasar panggilan fungsi, pastikan Anda melakukannya terlebih dahulu.
Dengan memanggil fungsi Anda dan parameternya dijelaskan ke API
sebagai genai.protos.FunctionDeclaration. Dalam kasus dasar, SDK bisa membangun
FunctionDeclaration dari fungsi dan anotasinya. Namun
Anda harus mendefinisikannya secara
eksplisit, untuk saat ini.
Menentukan skema
Mulai dengan menentukan person sebagai objek dengan kolom string name, description, start_place_name, end_place_name.
person = genai.protos.Schema(
type = genai.protos.Type.OBJECT,
properties = {
'name': genai.protos.Schema(type=genai.protos.Type.STRING),
'description': genai.protos.Schema(type=genai.protos.Type.STRING),
'start_place_name': genai.protos.Schema(type=genai.protos.Type.STRING),
'end_place_name': genai.protos.Schema(type=genai.protos.Type.STRING)
},
required=['name', 'description', 'start_place_name', 'end_place_name']
)
Kemudian, tentukan orang sebagai ARRAY dari objek person:
people = genai.protos.Schema(
type=genai.protos.Type.ARRAY,
items=person
)
Kemudian, lakukan hal yang sama untuk setiap entitas yang ingin Anda ekstrak:
place = genai.protos.Schema(
type = genai.protos.Type.OBJECT,
properties = {
'name': genai.protos.Schema(type=genai.protos.Type.STRING),
'description': genai.protos.Schema(type=genai.protos.Type.STRING),
}
)
places = genai.protos.Schema(
type=genai.protos.Type.ARRAY,
items=place
)
thing = genai.protos.Schema(
type = genai.protos.Type.OBJECT,
properties = {
'name': genai.protos.Schema(type=genai.protos.Type.STRING),
'description': genai.protos.Schema(type=genai.protos.Type.STRING),
}
)
things = genai.protos.Schema(
type=genai.protos.Type.ARRAY,
items=thing
)
relationship = genai.protos.Schema(
type = genai.protos.Type.OBJECT,
properties = {
'person_1_name': genai.protos.Schema(type=genai.protos.Type.STRING),
'person_2_name': genai.protos.Schema(type=genai.protos.Type.STRING),
'relationship': genai.protos.Schema(type=genai.protos.Type.STRING),
}
)
relationships = genai.protos.Schema(
type=genai.protos.Type.ARRAY,
items=relationship
)
Sekarang, bangun FunctionDeclaration:
add_to_database = genai.protos.FunctionDeclaration(
name="add_to_database",
description=textwrap.dedent("""\
Adds entities to the database.
"""),
parameters=genai.protos.Schema(
type=genai.protos.Type.OBJECT,
properties = {
'people': people,
'places': places,
'things': things,
'relationships': relationships
}
)
)
Memanggil API
Seperti yang Anda lihat di Dasar-dasar panggilan fungsi, kini Anda dapat meneruskan FunctionDeclaration ini ke argumen tools dari konstruktor genai.GenerativeModel (konstruktor juga akan menerima representasi JSON yang setara dari deklarasi fungsi):
model = genai.GenerativeModel(
model_name='models/gemini-1.5-pro-latest',
tools = [add_to_database])
Setiap kali Anda memanggil API, SDK akan mengirim alat beserta perintah Anda, dan model akan memanggil fungsi yang telah Anda tetapkan:
result = model.generate_content(f"""
Please add the people, places, things, and relationships from this story to the database:
{story}
""",
# Force a function call
tool_config={'function_calling_config':'ANY'})
Sekarang tidak ada teks untuk diurai. Hasilnya adalah struktur data.
'text' in result.candidates[0].content.parts[0]
False
'function_call' in result.candidates[0].content.parts[0]
True
fc = result.candidates[0].content.parts[0].function_call
print(type(fc))
<class 'google.ai.generativelanguage_v1beta.types.content.FunctionCall'>
Class genai.protos.FunctionCall didasarkan pada Buffering Protokol Google,
mengubahnya menjadi objek kompatibel dengan JSON yang lebih dikenal:
print(json.dumps(type(fc).to_dict(fc), indent=4))
{
"name": "add_to_database",
"args": {
"things": [
{
"name": "Magical Backpack",
"description": "Anya's prized possession, the Magical Backpack, is no ordinary satchel. Its soft, emerald-green fabric shimmers with an ethereal glow, and its leather straps have secrets that only Anya knows. Within its capacious interior lay an enchanted world, filled with wonders that would ignite her imagination and change her life forever."
},
{
"name": "Shimmering Sword",
"description": "Among the wonders in Anya's Magical Backpack, lies a shimmering sword. With a determined gleam in her eye, she retrieved the shimmering sword and charged towards the monster."
},
{
"description": "Residing within the Magical Backpack, the Book of Ancient Spells holds secrets untold.",
"name": "Book of Ancient Spells"
},
{
"description": "Tucked away in the Magical Backpack is a tiny compass that always points north.",
"name": "Tiny Compass that Always Points North"
},
{
"description": "Hidden within the Magical Backpack is a magical key that can open any lock.",
"name": "Magical Key that Can Open Any Lock"
}
],
"relationships": [
{
"relationship": "Mother-Daughter",
"person_1_name": "Anya",
"person_2_name": "Elise"
},
{
"person_2_name": "Edward",
"relationship": "Father-Daughter",
"person_1_name": "Anya"
},
{
"person_2_name": "Samuel",
"person_1_name": "Anya",
"relationship": "Best Friends"
}
],
"people": [
{
"name": "Anya",
"description": "Anya, the main character of the story, is a young girl with a magical backpack.",
"start_place_name": "Willow Creek",
"end_place_name": "Unknown"
},
{
"name": "Elise",
"description": "Anya's mother, Elise is a kind-hearted woman.",
"end_place_name": "Unknown",
"start_place_name": "Willow Creek"
},
{
"start_place_name": "Willow Creek",
"end_place_name": "Unknown",
"name": "Edward",
"description": "Anya's father, Edward is a wise-bearded man."
},
{
"end_place_name": "Unknown",
"start_place_name": "Willow Creek",
"description": "Anya's best friend, Samuel is a curious and adventurous boy with a mischievous grin.",
"name": "Samuel"
}
],
"places": [
{
"description": "The quaint town of Willow Creek is nestled amidst rolling hills and whispering willows.",
"name": "Willow Creek"
},
{
"description": "The town's only schoolhouse.",
"name": "Schoolhouse"
},
{
"description": "A shadowy place filled with secrets and dangers, the Forest is home to a terrifying monster.",
"name": "Forest"
}
]
}
}
Kesimpulan
Meskipun API dapat menangani masalah ekstraksi data terstruktur dengan input teks murni dan output teks, penggunaan panggilan fungsi kemungkinan lebih andal karena memungkinkan Anda menentukan skema yang ketat, dan menghilangkan langkah penguraian yang berpotensi rentan error.