Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4510 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.5308. Min Act: -0.0000

Data Index: 1577476 (Python Code)

Max Activating Token Index: 639

Text #1

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.6547. Min Act: -0.0000

Data Index: 1443339 (Python Code)

Max Activating Token Index: 362

Text #2

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.6418. Min Act: -0.0000

Data Index: 1644708 (Python Code)

Max Activating Token Index: 1010

Text #3

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.6172. Min Act: -0.0000

Data Index: 1563111 (Python Code)

Max Activating Token Index: 62

Text #4

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.3561. Min Act: -0.0000

Data Index: 1539417 (Python Code)

Max Activating Token Index: 377

Text #5

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.3584. Min Act: -0.0000

Data Index: 1452083 (Python Code)

Max Activating Token Index: 337

Text #6

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.3184. Min Act: -0.0000

Data Index: 1631482 (Python Code)

Max Activating Token Index: 304

Text #7

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.0289. Min Act: -0.0000

Data Index: 1379567 (Python Code)

Max Activating Token Index: 959

Text #8

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.1682. Min Act: -0.0000

Data Index: 1445847 (Python Code)

Max Activating Token Index: 377

Text #9

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.1501. Min Act: -0.0000

Data Index: 1400296 (Python Code)

Max Activating Token Index: 445

Text #10

Max Range: 4.6547. Min Range: -4.6547

Max Act: 3.9852. Min Act: -0.0000

Data Index: 1438433 (Python Code)

Max Activating Token Index: 1008

Text #11

Max Range: 4.6547. Min Range: -4.6547

Max Act: 3.9572. Min Act: -0.0000

Data Index: 1374491 (Python Code)

Max Activating Token Index: 706

Text #12

Max Range: 4.6547. Min Range: -4.6547

Max Act: 3.9400. Min Act: -0.0000

Data Index: 1530077 (Python Code)

Max Activating Token Index: 389

Text #13

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.0955. Min Act: -0.0000

Data Index: 1650877 (Python Code)

Max Activating Token Index: 563

Text #14

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.0948. Min Act: -0.0000

Data Index: 1451646 (Python Code)

Max Activating Token Index: 232

Text #15

Max Range: 4.6547. Min Range: -4.6547

Max Act: 4.2457. Min Act: -0.0000

Data Index: 1532405 (Python Code)

Max Activating Token Index: 212

Text #16

Max Range: 4.6547. Min Range: -4.6547

Max Act: 3.8787. Min Act: -0.0000

Data Index: 1498698 (Python Code)

Max Activating Token Index: 277

Text #17

Max Range: 4.6547. Min Range: -4.6547

Max Act: 3.9569. Min Act: -0.0000

Data Index: 1530385 (Python Code)

Max Activating Token Index: 396

Text #18

Max Range: 4.6547. Min Range: -4.6547

Max Act: 3.9261. Min Act: -0.0000

Data Index: 1504078 (Python Code)

Max Activating Token Index: 215

Text #19

Max Range: 4.6547. Min Range: -4.6547

Max Act: 3.7590. Min Act: -0.0000

Data Index: 1499679 (Python Code)

Max Activating Token Index: 407