Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2987 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.3558. Min Act: -0.0001

Data Index: 601004 (C4 (Web Text))

Max Activating Token Index: 928

Text #1

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.3531. Min Act: -0.0001

Data Index: 624545 (C4 (Web Text))

Max Activating Token Index: 568

Text #2

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.3424. Min Act: -0.0000

Data Index: 33073 (C4 (Web Text))

Max Activating Token Index: 258

Text #3

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.3304. Min Act: -0.0001

Data Index: 377980 (C4 (Web Text))

Max Activating Token Index: 935

Text #4

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.3125. Min Act: -0.0001

Data Index: 698147 (C4 (Web Text))

Max Activating Token Index: 895

Text #5

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2748. Min Act: -0.0001

Data Index: 1309611 (C4 (Web Text))

Max Activating Token Index: 634

Text #6

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2788. Min Act: -0.0001

Data Index: 24753 (C4 (Web Text))

Max Activating Token Index: 485

Text #7

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2999. Min Act: -0.0001

Data Index: 326681 (C4 (Web Text))

Max Activating Token Index: 645

Text #8

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2730. Min Act: -0.0000

Data Index: 1203167 (C4 (Web Text))

Max Activating Token Index: 391

Text #9

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2636. Min Act: -0.0001

Data Index: 976327 (C4 (Web Text))

Max Activating Token Index: 912

Text #10

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2586. Min Act: -0.0001

Data Index: 1161998 (C4 (Web Text))

Max Activating Token Index: 777

Text #11

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2656. Min Act: -0.0001

Data Index: 783876 (C4 (Web Text))

Max Activating Token Index: 764

Text #12

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2427. Min Act: -0.0000

Data Index: 1329696 (C4 (Web Text))

Max Activating Token Index: 422

Text #13

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2318. Min Act: -0.0001

Data Index: 1144570 (C4 (Web Text))

Max Activating Token Index: 627

Text #14

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2403. Min Act: -0.0000

Data Index: 1116506 (C4 (Web Text))

Max Activating Token Index: 462

Text #15

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2394. Min Act: -0.0001

Data Index: 56964 (C4 (Web Text))

Max Activating Token Index: 381

Text #16

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2272. Min Act: -0.0000

Data Index: 1091887 (C4 (Web Text))

Max Activating Token Index: 202

Text #17

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2254. Min Act: -0.0000

Data Index: 565499 (C4 (Web Text))

Max Activating Token Index: 730

Text #18

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2302. Min Act: -0.0001

Data Index: 791313 (C4 (Web Text))

Max Activating Token Index: 608

Text #19

Max Range: 0.3558. Min Range: -0.3558

Max Act: 0.2314. Min Act: -0.0001

Data Index: 375633 (C4 (Web Text))

Max Activating Token Index: 410