Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5593 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.4911. Min Act: -0.0000

Data Index: 677512 (C4 (Web Text))

Max Activating Token Index: 412

Text #1

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.5130. Min Act: -0.0000

Data Index: 249860 (C4 (Web Text))

Max Activating Token Index: 899

Text #2

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.3377. Min Act: -0.0000

Data Index: 943604 (C4 (Web Text))

Max Activating Token Index: 718

Text #3

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.3425. Min Act: -0.0000

Data Index: 1090827 (C4 (Web Text))

Max Activating Token Index: 627

Text #4

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.2139. Min Act: -0.0000

Data Index: 1208177 (C4 (Web Text))

Max Activating Token Index: 769

Text #5

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.2696. Min Act: -0.0000

Data Index: 398543 (C4 (Web Text))

Max Activating Token Index: 220

Text #6

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.2093. Min Act: -0.0000

Data Index: 740592 (C4 (Web Text))

Max Activating Token Index: 685

Text #7

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.2795. Min Act: -0.0000

Data Index: 1094141 (C4 (Web Text))

Max Activating Token Index: 126

Text #8

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.2691. Min Act: -0.0000

Data Index: 469003 (C4 (Web Text))

Max Activating Token Index: 287

Text #9

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.1305. Min Act: -0.0000

Data Index: 86752 (C4 (Web Text))

Max Activating Token Index: 848

Text #10

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.2789. Min Act: -0.0000

Data Index: 386138 (C4 (Web Text))

Max Activating Token Index: 209

Text #11

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.1858. Min Act: -0.0000

Data Index: 278351 (C4 (Web Text))

Max Activating Token Index: 791

Text #12

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.1170. Min Act: -0.0000

Data Index: 50362 (C4 (Web Text))

Max Activating Token Index: 460

Text #13

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.1863. Min Act: -0.0000

Data Index: 711468 (C4 (Web Text))

Max Activating Token Index: 149

Text #14

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.2631. Min Act: -0.0000

Data Index: 105280 (C4 (Web Text))

Max Activating Token Index: 372

Text #15

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.1813. Min Act: -0.0000

Data Index: 195934 (C4 (Web Text))

Max Activating Token Index: 926

Text #16

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.1521. Min Act: -0.0000

Data Index: 515008 (C4 (Web Text))

Max Activating Token Index: 634

Text #17

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.2300. Min Act: -0.0000

Data Index: 524891 (C4 (Web Text))

Max Activating Token Index: 322

Text #18

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.1445. Min Act: -0.0000

Data Index: 345780 (C4 (Web Text))

Max Activating Token Index: 447

Text #19

Max Range: 1.5130. Min Range: -1.5130

Max Act: 1.2041. Min Act: -0.0000

Data Index: 898002 (C4 (Web Text))

Max Activating Token Index: 365