Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4947 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.2466. Min Range: -5.2466

Max Act: 5.2466. Min Act: 0.0000

Data Index: 1375955 (Python Code)

Max Activating Token Index: 246

Text #1

Max Range: 5.2466. Min Range: -5.2466

Max Act: 5.0479. Min Act: -0.0000

Data Index: 1527559 (Python Code)

Max Activating Token Index: 309

Text #2

Max Range: 5.2466. Min Range: -5.2466

Max Act: 5.0846. Min Act: -0.0000

Data Index: 1523436 (Python Code)

Max Activating Token Index: 233

Text #3

Max Range: 5.2466. Min Range: -5.2466

Max Act: 5.0013. Min Act: -0.0000

Data Index: 1548399 (Python Code)

Max Activating Token Index: 645

Text #4

Max Range: 5.2466. Min Range: -5.2466

Max Act: 5.1349. Min Act: -0.0000

Data Index: 1392255 (Python Code)

Max Activating Token Index: 365

Text #5

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.7670. Min Act: -0.0000

Data Index: 1550573 (Python Code)

Max Activating Token Index: 531

Text #6

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.9368. Min Act: -0.0000

Data Index: 1642716 (Python Code)

Max Activating Token Index: 43

Text #7

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.7380. Min Act: -0.0000

Data Index: 1624928 (Python Code)

Max Activating Token Index: 571

Text #8

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.9087. Min Act: -0.0000

Data Index: 1451658 (Python Code)

Max Activating Token Index: 490

Text #9

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.6652. Min Act: -0.0000

Data Index: 1509458 (Python Code)

Max Activating Token Index: 228

Text #10

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.8032. Min Act: -0.0000

Data Index: 1378020 (Python Code)

Max Activating Token Index: 539

Text #11

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.9285. Min Act: -0.0000

Data Index: 1374242 (Python Code)

Max Activating Token Index: 361

Text #12

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.9134. Min Act: 0.0000

Data Index: 1368996 (Python Code)

Max Activating Token Index: 538

Text #13

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.5501. Min Act: -0.0000

Data Index: 1600473 (Python Code)

Max Activating Token Index: 536

Text #14

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.8534. Min Act: -0.0000

Data Index: 1549390 (Python Code)

Max Activating Token Index: 326

Text #15

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.5144. Min Act: -0.0000

Data Index: 1570176 (Python Code)

Max Activating Token Index: 320

Text #16

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.8422. Min Act: -0.0000

Data Index: 1530834 (Python Code)

Max Activating Token Index: 37

Text #17

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.6419. Min Act: -0.0000

Data Index: 1405614 (Python Code)

Max Activating Token Index: 824

Text #18

Max Range: 5.2466. Min Range: -5.2466

Max Act: 4.6551. Min Act: -0.0000

Data Index: 1416505 (Python Code)

Max Activating Token Index: 162

Text #19

Max Range: 5.2466. Min Range: -5.2466

Max Act: 5.0189. Min Act: -0.0000

Data Index: 1452462 (Python Code)

Max Activating Token Index: 273