Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3024 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.3290. Min Act: -0.0001

Data Index: 1435098 (Python Code)

Max Activating Token Index: 406

Text #1

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.3075. Min Act: -0.0001

Data Index: 1423268 (Python Code)

Max Activating Token Index: 702

Text #2

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2966. Min Act: -0.0001

Data Index: 1617309 (Python Code)

Max Activating Token Index: 544

Text #3

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2964. Min Act: -0.0001

Data Index: 1408350 (Python Code)

Max Activating Token Index: 324

Text #4

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.3058. Min Act: -0.0001

Data Index: 1604228 (Python Code)

Max Activating Token Index: 582

Text #5

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.3155. Min Act: -0.0001

Data Index: 476472 (C4 (Web Text))

Max Activating Token Index: 608

Text #6

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2892. Min Act: -0.0001

Data Index: 1512723 (Python Code)

Max Activating Token Index: 818

Text #7

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2706. Min Act: -0.0001

Data Index: 1360757 (Python Code)

Max Activating Token Index: 112

Text #8

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2708. Min Act: -0.0001

Data Index: 1004330 (C4 (Web Text))

Max Activating Token Index: 854

Text #9

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2966. Min Act: -0.0001

Data Index: 1126572 (C4 (Web Text))

Max Activating Token Index: 783

Text #10

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2764. Min Act: -0.0001

Data Index: 81045 (C4 (Web Text))

Max Activating Token Index: 625

Text #11

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2814. Min Act: -0.0001

Data Index: 853632 (C4 (Web Text))

Max Activating Token Index: 96

Text #12

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2629. Min Act: -0.0001

Data Index: 1462934 (Python Code)

Max Activating Token Index: 494

Text #13

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2454. Min Act: -0.0001

Data Index: 948742 (C4 (Web Text))

Max Activating Token Index: 823

Text #14

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2670. Min Act: -0.0001

Data Index: 1460210 (Python Code)

Max Activating Token Index: 288

Text #15

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2609. Min Act: -0.0001

Data Index: 867334 (C4 (Web Text))

Max Activating Token Index: 169

Text #16

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2437. Min Act: -0.0001

Data Index: 1400998 (Python Code)

Max Activating Token Index: 223

Text #17

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2518. Min Act: -0.0001

Data Index: 1546037 (Python Code)

Max Activating Token Index: 290

Text #18

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2511. Min Act: -0.0001

Data Index: 1425148 (Python Code)

Max Activating Token Index: 647

Text #19

Max Range: 0.3290. Min Range: -0.3290

Max Act: 0.2407. Min Act: -0.0001

Data Index: 642413 (C4 (Web Text))

Max Activating Token Index: 548