Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 128 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.4855. Min Act: -0.0000

Data Index: 1636629 (Python Code)

Max Activating Token Index: 882

Text #1

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.4798. Min Act: -0.0000

Data Index: 1650135 (Python Code)

Max Activating Token Index: 605

Text #2

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.3069. Min Act: -0.0000

Data Index: 1585631 (Python Code)

Max Activating Token Index: 157

Text #3

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.3240. Min Act: -0.0000

Data Index: 1494221 (Python Code)

Max Activating Token Index: 375

Text #4

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.2043. Min Act: -0.0000

Data Index: 1461765 (Python Code)

Max Activating Token Index: 186

Text #5

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.1498. Min Act: -0.0000

Data Index: 1374378 (Python Code)

Max Activating Token Index: 256

Text #6

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.0016. Min Act: -0.0000

Data Index: 1452920 (Python Code)

Max Activating Token Index: 363

Text #7

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.1135. Min Act: -0.0000

Data Index: 1601752 (Python Code)

Max Activating Token Index: 653

Text #8

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.3348. Min Act: -0.0000

Data Index: 1483007 (Python Code)

Max Activating Token Index: 161

Text #9

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.2290. Min Act: -0.0000

Data Index: 1599036 (Python Code)

Max Activating Token Index: 834

Text #10

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.3229. Min Act: -0.0000

Data Index: 1602565 (Python Code)

Max Activating Token Index: 61

Text #11

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.0888. Min Act: -0.0000

Data Index: 1483418 (Python Code)

Max Activating Token Index: 112

Text #12

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.2036. Min Act: -0.0000

Data Index: 1610256 (Python Code)

Max Activating Token Index: 72

Text #13

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.0606. Min Act: -0.0000

Data Index: 1563130 (Python Code)

Max Activating Token Index: 741

Text #14

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.1261. Min Act: -0.0000

Data Index: 1438790 (Python Code)

Max Activating Token Index: 534

Text #15

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.0929. Min Act: -0.0000

Data Index: 1413358 (Python Code)

Max Activating Token Index: 57

Text #16

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.0832. Min Act: -0.0000

Data Index: 1578434 (Python Code)

Max Activating Token Index: 188

Text #17

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.2039. Min Act: -0.0000

Data Index: 1492307 (Python Code)

Max Activating Token Index: 67

Text #18

Max Range: 2.4855. Min Range: -2.4855

Max Act: 2.0405. Min Act: -0.0000

Data Index: 1612953 (Python Code)

Max Activating Token Index: 957

Text #19

Max Range: 2.4855. Min Range: -2.4855

Max Act: 1.9824. Min Act: -0.0000

Data Index: 1570432 (Python Code)

Max Activating Token Index: 845