Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4871 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.7902. Min Act: -0.0000

Data Index: 1520685 (Python Code)

Max Activating Token Index: 660

Text #1

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.8174. Min Act: -0.0000

Data Index: 1410830 (Python Code)

Max Activating Token Index: 187

Text #2

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.6500. Min Act: -0.0000

Data Index: 1650904 (Python Code)

Max Activating Token Index: 797

Text #3

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.7504. Min Act: -0.0000

Data Index: 1503597 (Python Code)

Max Activating Token Index: 851

Text #4

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.5965. Min Act: -0.0000

Data Index: 1442768 (Python Code)

Max Activating Token Index: 681

Text #5

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.5657. Min Act: -0.0000

Data Index: 1452297 (Python Code)

Max Activating Token Index: 730

Text #6

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.5413. Min Act: -0.0000

Data Index: 1415233 (Python Code)

Max Activating Token Index: 414

Text #7

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.6308. Min Act: -0.0000

Data Index: 1560644 (Python Code)

Max Activating Token Index: 521

Text #8

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.5322. Min Act: -0.0000

Data Index: 1453897 (Python Code)

Max Activating Token Index: 42

Text #9

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.4936. Min Act: -0.0000

Data Index: 1543317 (Python Code)

Max Activating Token Index: 339

Text #10

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.4449. Min Act: -0.0000

Data Index: 1526066 (Python Code)

Max Activating Token Index: 889

Text #11

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.3842. Min Act: -0.0000

Data Index: 946033 (C4 (Web Text))

Max Activating Token Index: 125

Text #12

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.3146. Min Act: -0.0000

Data Index: 601968 (C4 (Web Text))

Max Activating Token Index: 19

Text #13

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.3623. Min Act: -0.0000

Data Index: 1508904 (Python Code)

Max Activating Token Index: 1014

Text #14

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.1561. Min Act: -0.0000

Data Index: 211553 (C4 (Web Text))

Max Activating Token Index: 746

Text #15

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.0804. Min Act: -0.0000

Data Index: 665240 (C4 (Web Text))

Max Activating Token Index: 142

Text #16

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.1460. Min Act: -0.0000

Data Index: 415994 (C4 (Web Text))

Max Activating Token Index: 635

Text #17

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.0804. Min Act: -0.0000

Data Index: 1541026 (Python Code)

Max Activating Token Index: 419

Text #18

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.0597. Min Act: -0.0000

Data Index: 1516843 (Python Code)

Max Activating Token Index: 70

Text #19

Max Range: 1.8174. Min Range: -1.8174

Max Act: 1.1132. Min Act: -0.0000

Data Index: 1648704 (Python Code)

Max Activating Token Index: 146