Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5830 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.3135. Min Range: -3.3135

Max Act: 3.3135. Min Act: -0.0000

Data Index: 1545909 (Python Code)

Max Activating Token Index: 856

Text #1

Max Range: 3.3135. Min Range: -3.3135

Max Act: 3.2756. Min Act: -0.0000

Data Index: 1543296 (Python Code)

Max Activating Token Index: 461

Text #2

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.8954. Min Act: -0.0000

Data Index: 1606628 (Python Code)

Max Activating Token Index: 787

Text #3

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.7009. Min Act: -0.0000

Data Index: 1515491 (Python Code)

Max Activating Token Index: 526

Text #4

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.6258. Min Act: -0.0000

Data Index: 1556331 (Python Code)

Max Activating Token Index: 473

Text #5

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.6311. Min Act: -0.0000

Data Index: 1654020 (Python Code)

Max Activating Token Index: 969

Text #6

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.5700. Min Act: -0.0000

Data Index: 1439495 (Python Code)

Max Activating Token Index: 1021

Text #7

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.3730. Min Act: -0.0000

Data Index: 1617169 (Python Code)

Max Activating Token Index: 169

Text #8

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.4346. Min Act: -0.0000

Data Index: 1561463 (Python Code)

Max Activating Token Index: 997

Text #9

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.4420. Min Act: -0.0000

Data Index: 1402812 (Python Code)

Max Activating Token Index: 62

Text #10

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.4294. Min Act: -0.0000

Data Index: 1633976 (Python Code)

Max Activating Token Index: 82

Text #11

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.4395. Min Act: -0.0000

Data Index: 1554458 (Python Code)

Max Activating Token Index: 425

Text #12

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.2787. Min Act: -0.0000

Data Index: 1513507 (Python Code)

Max Activating Token Index: 945

Text #13

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.2374. Min Act: -0.0000

Data Index: 1503001 (Python Code)

Max Activating Token Index: 576

Text #14

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.3575. Min Act: -0.0000

Data Index: 1460377 (Python Code)

Max Activating Token Index: 97

Text #15

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.0923. Min Act: -0.0000

Data Index: 1652770 (Python Code)

Max Activating Token Index: 230

Text #16

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.3106. Min Act: -0.0000

Data Index: 1653781 (Python Code)

Max Activating Token Index: 103

Text #17

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.3060. Min Act: -0.0000

Data Index: 1421524 (Python Code)

Max Activating Token Index: 95

Text #18

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.1783. Min Act: -0.0000

Data Index: 1621893 (Python Code)

Max Activating Token Index: 33

Text #19

Max Range: 3.3135. Min Range: -3.3135

Max Act: 2.1653. Min Act: -0.0000

Data Index: 1475040 (Python Code)

Max Activating Token Index: 999