Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 108 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0614. Min Range: -3.0614

Max Act: 3.0614. Min Act: -0.0000

Data Index: 1604550 (Python Code)

Max Activating Token Index: 158

Text #1

Max Range: 3.0614. Min Range: -3.0614

Max Act: 2.4042. Min Act: -0.0000

Data Index: 1483277 (Python Code)

Max Activating Token Index: 621

Text #2

Max Range: 3.0614. Min Range: -3.0614

Max Act: 2.3273. Min Act: -0.0000

Data Index: 1542421 (Python Code)

Max Activating Token Index: 860

Text #3

Max Range: 3.0614. Min Range: -3.0614

Max Act: 2.1316. Min Act: -0.0000

Data Index: 1566676 (Python Code)

Max Activating Token Index: 595

Text #4

Max Range: 3.0614. Min Range: -3.0614

Max Act: 2.3090. Min Act: -0.0000

Data Index: 1648103 (Python Code)

Max Activating Token Index: 855

Text #5

Max Range: 3.0614. Min Range: -3.0614

Max Act: 2.1818. Min Act: -0.0000

Data Index: 1649942 (Python Code)

Max Activating Token Index: 159

Text #6

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.9635. Min Act: -0.0000

Data Index: 1409434 (Python Code)

Max Activating Token Index: 899

Text #7

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.9810. Min Act: -0.0000

Data Index: 1362759 (Python Code)

Max Activating Token Index: 852

Text #8

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.9967. Min Act: -0.0000

Data Index: 1366766 (Python Code)

Max Activating Token Index: 713

Text #9

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.8685. Min Act: -0.0000

Data Index: 1507028 (Python Code)

Max Activating Token Index: 466

Text #10

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.7805. Min Act: -0.0000

Data Index: 1466819 (Python Code)

Max Activating Token Index: 984

Text #11

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.9257. Min Act: -0.0000

Data Index: 1420882 (Python Code)

Max Activating Token Index: 693

Text #12

Max Range: 3.0614. Min Range: -3.0614

Max Act: 2.0359. Min Act: -0.0000

Data Index: 1647958 (Python Code)

Max Activating Token Index: 987

Text #13

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.6855. Min Act: -0.0000

Data Index: 1579721 (Python Code)

Max Activating Token Index: 788

Text #14

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.6765. Min Act: -0.0000

Data Index: 1494141 (Python Code)

Max Activating Token Index: 712

Text #15

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.9544. Min Act: -0.0000

Data Index: 1556534 (Python Code)

Max Activating Token Index: 785

Text #16

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.6427. Min Act: -0.0000

Data Index: 1583183 (Python Code)

Max Activating Token Index: 713

Text #17

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.6787. Min Act: -0.0000

Data Index: 1424304 (Python Code)

Max Activating Token Index: 254

Text #18

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.8572. Min Act: -0.0000

Data Index: 1568404 (Python Code)

Max Activating Token Index: 652

Text #19

Max Range: 3.0614. Min Range: -3.0614

Max Act: 1.8517. Min Act: -0.0000

Data Index: 1564592 (Python Code)

Max Activating Token Index: 303